In the field of IR, natural language processing is one of the most important tasks that an automated system must perform. The filtration & tokenization of text in particular is of great importance in order to be able to mathematically represent, classify and ultimately analyze the given text in a document. As this technology becomes more and more advanced, we can expect to see higher and higher quality content moving up in rankings at the search engines.
For those of us who aren’t academics in the field of natural language processing, all of this can seem quite daunting. It’s often hard to even understand how the process works or what our documents are faced with when the search engines conduct text analysis on them. Luckily, I’ve come across a downloadable project at SourceForge that provides a basic overview and a software application for natural langauge processing.
If you’re interested in learning more, I’d urge you to:
- Read the overview, which discusses the who, what and how of the project
- Go through the documentation. Even if you don’t grasp all of the programming and math, the text will walk you through examples of exactly what is being performed by the software and why.
- If you’re bold enough to want more, read the instructions for installation and give it a go – don’t forget you’ll need the Python Toolkit installed on your server to make this work.
Even a brief overview of how the system works can provide great insight into how a computer processes text. It’s also a very convincing argument for why keyword density is virtually non-existant in document analysis at the search engines.
